Primary modeling
We had two primary outcomes of interest: CIS and PIO4. The CIS scores measure career interest and PIO4 measures STEM identity. For each outcome, we fit a two-level cross-classified hierarchical linear model (HLM) with Level 1 individuals (\(i = 1,2, \dots, n_j\)) nested within Level 2 classes (\(j = 1,2, \dots J\)). Our data structure is considered cross-classified since the same student may belong to more than one class (e.g. take BIOL 151 and CHEM 151 in the same semester). Each model was of the following form:
Level 1: students
\[\text{postscore}_{ij} = \beta_0 + \beta_1 (\text{prescore}_{ij} - \overline{\text{prescore}}_j)_{ij} + \beta_2X_{2_{ij}} + \cdots \beta_pX_{p_{ij}}+\epsilon_{ij} \]
Level 2: classes
\[\beta_0 = \gamma_{00} + \gamma_{01}\text{LA}_j + \gamma_{02}\text{GE}_j + \gamma_{03}\text{LA*GE}_j + \gamma_{04}\overline{\text{prescore}}_j + \gamma_{05}W_{5_j} + \cdots + \gamma_{0q}W_{q_j} + u_{j}\]
\[\beta_1 = \gamma_{10}\] \[\vdots\] \[\beta_p = \gamma_{p0}\]
Combined:
\[y_{ij} = \gamma_{00} + \gamma_{01}\text{LA}_j + \gamma_{02}\text{GE}_j + \gamma_{03}\text{LA*GE}_j + \gamma_{04}\overline{\text{prescore}}_j + \gamma_{05}W_{5_j} + \cdots + \gamma_{0q}W_{q_j} + \gamma_{10} (\text{prescore}_{ij} - \overline{\text{prescore}}_j)_{ij} + \gamma_{20}X_{2_{ij}} + \cdots \gamma_{p0}X_{p_{ij}} + \epsilon_{ij} + u_j\]
Note, the following specifications of the model:
- prescores are group mean centered at level 1, and the group mean is also included as a level-2 predictor. This allows for both within- and between-group variation in prescores to be accounted for in the model.
- \(X\) denotes a generic level-1 predictor
- \(W\) denotes a generic level-2 predictor
- \(p\) denotes the number of level-1 predictors
- \(q\) denotes the number of level-2 predictors
- the treatment variable of primary interest is the level-2 variable \(LA\), which is an indicator variable for whether the class had an LA or not
- the treatment variable is interacted with GE, a level-2 variable indicating whether the class was a General Education course or a “STEM Major Gateway” course. We expect the outcomes (STEM career interest & identity) to behave differently in these different course types, and this was also used as a stratifying variable in the research plan when assigning treatment (LA) and control (no LA) status to each course. Therefore, the GE variable and its interaction with LA is included in every model, regardless of statistical significance.
- \(\epsilon_{ij}\) is the level-1 (individual) error term, assumed to be normally distributed
- \(\u_j\) is the level-2 (class) error term, assumed to be normally distributed
| term | estimate | std.error | statistic | df | p.value |
|---|---|---|---|---|---|
| (Intercept) | 10.3637091 | 7.0701381 | 1.4658425 | 190.46597 | 0.1443401 |
| score_CIS_PRE_grp_cntr | 0.6558413 | 0.0560807 | 11.6945998 | 63.20520 | 0.0000000 |
| score_CIS_PRE_grp_avg | 0.7572481 | 0.1568356 | 4.8282928 | 194.04150 | 0.0000028 |
| ethnicity_collapsedAsian | 2.0878239 | 0.8383434 | 2.4904160 | 115.79179 | 0.0141779 |
| ethnicity_collapsedWhite | 0.8603214 | 0.6383994 | 1.3476225 | 111.17314 | 0.1805197 |
| ethnicity_collapsedOther | 1.6770966 | 0.9940697 | 1.6871016 | 43.12047 | 0.0988062 |
| stem_majorYes | 1.4514459 | 0.6648323 | 2.1831761 | 94.82581 | 0.0314911 |
| prop_stem_major_grd_cntr | -4.9043843 | 2.8682052 | -1.7099140 | 275.40758 | 0.0884081 |
| grade_ABCB | -0.8525435 | 0.5410869 | -1.5756130 | 175.02184 | 0.1169208 |
| grade_ABCC | -1.4604493 | 0.7377179 | -1.9796854 | 119.95977 | 0.0500287 |
| grade_ABCD | -3.4660689 | 0.9044460 | -3.8322565 | 208.23528 | 0.0001681 |
| grade_ABCF | -3.4636456 | 1.0556696 | -3.2809940 | 174.01301 | 0.0012498 |
| grade_ABCW | -2.4961548 | 1.0975515 | -2.2742940 | 104.59269 | 0.0249896 |
| subjectCHEM | 2.8081829 | 1.4369522 | 1.9542633 | 257.94937 | 0.0517505 |
| subjectMATH | 3.0446094 | 1.4840378 | 2.0515713 | 256.80153 | 0.0412256 |
| subjectPHYC | 2.1204187 | 1.4075154 | 1.5064977 | 283.64272 | 0.1330523 |
| LA_classyes | -1.2417809 | 1.2928871 | -0.9604713 | 269.60192 | 0.3376787 |
| gate_geGE | -4.5358487 | 2.2940811 | -1.9771963 | 221.10967 | 0.0492618 |
| LA_classyes:gate_geGE | 0.3305614 | 2.1737311 | 0.1520710 | 244.46153 | 0.8792564 |
Interpretations:
Note the intercept represents the average end-of-semester CIS-score for a very specific (possibly non-existent) respondent: a Hispanic non-STEM major who received an A in their non-LA STEM gateway BIOL course and whose pre-score was average (relative to other students in their course) and whose course’s pre-score average was average relative to all other courses.
The positive coefficient for
ethnicity_collapsedAsiansuggests that on average, an Asian student will have a higher post-CIS score (+2.09) compared to a Hispanic student in the same course who received the same grade and had the same pre-score and STEM major statusThe positive coefficient for
stem_majorYessuggests that on average, a STEM major will have a higher post-CIS score (+1.45) compared to a non-STEM major of the same ethnicity in the same course who received the same grade and had the same pre-scoreThe negative coefficient for
grade_ABCDDsuggests that on average, a student who receives a D in their 100-level STEM course will have a lower post-CIS score (–3.47) compared to a student who receives an A in the same course, even when controlling for pre-score, ethnicity, STEM major status, and course characteristicsThe negative coefficient for
grade_ABCDFsuggests that on average, a student who receives a F in their 100-level STEM course will have a lower post-CIS score (–3.46) compared to a student who receives an A in the same course, even when controlling for pre-score, ethnicity, STEM major status, and course characteristicsThe negative coefficient for
grade_ABCDWsuggests that on average, a student who withdraws from their 100-level STEM course will have a lower post-CIS score (–2.5) compared to a student who receives an A in the same course, even when controlling for pre-score, ethnicity, STEM major status, and course characteristicsThe positive coefficients for
subjetCHEM,subjectMATH, andsubjectPHYCsuggest that on average, a student in a Chemistry, Math, or Physics course, respectively, will have a higher post-CIS score (+2.81, +2.81, ) compared to a similar student (same pre-score, ethnicity, and course grade) in a similar Biology course (same pre-score average, proportion of STEM majors in the course, LA status, and GE/Gateway status). Note that only the difference between Math and Biology is statistically significant, however.The negative coefficient for
gate_geGEsuggests that on average, a student in a 100-level STEM GE course will have a lower post-CIS score (–4.54) compared to a similar student (same pre-score, ethnicity, and course grade) in a similar 100-level STEM Gateway course (same pre-score average, proportion of STEM majors in the course, LA status, and subject (BIOL/CHEM/MATH/PHYS))Pre-scores are strong predictors of post-scores, as is to be expected. All other variables in the model are not statistically significant, including the treatment indicator for whether or not there was an LA in the class. That is, there is not sufficient evidence to suggest that having an LA in a course improves student STEM Career Interest (in fact the coefficient is negative, albeit non-significant).